Fix random fails in test_commondata#2474
Conversation
Yes, please do! This is important
If they seem to do the same thing (and given that you will most likely have a script to check whether the covmat is the same given the point above) just remove one and check.
Because we are evil and deserve divine punishment. Please remove one for salvation.
How often does it happen? My naive guess is that float precision + ordering might take care of this edge case as well (up to updates of numpy/scipy that might change the order of operations) |
This comment was marked as spam.
This comment was marked as spam.
Sorry, something went wrong.
|
What is the status of this? https://github.com/NNPDF/nnpdf/actions/runs/26394014558/job/77690240148 The ratio of fail/non fail is getting close to 50% now 😢 |
|
I have been very busy with my annual report the last few days, I'll get back to this PR soon! |
3 participants
As per #2444,
test_commondatasometimes randomly fails due to a worker with a different version of some package.Example list of failing datasets
ATLAS_2JET_7TEV_R06/uncertainties.yaml ATLAS_2JET_7TEV_R06/uncertainties_stronger.yaml ATLAS_2JET_7TEV_R06/uncertainties_weaker.yaml ATLAS_TTBAR_13TEV_HADR_DIF/uncertainties_dSig_dyttBar_norm.yaml ATLAS_TTBAR_13TEV_LJ_DIF/uncertainties_dSig_dmttBar_interspectra.yaml ATLAS_TTBAR_13TEV_LJ_DIF/uncertainties_dSig_dmttBar_norm.yaml ATLAS_TTBAR_13TEV_LJ_DIF/uncertainties_dSig_dpTt_interspectra.yaml ATLAS_TTBAR_13TEV_LJ_DIF/uncertainties_dSig_dyt_interspectra.yaml ATLAS_TTBAR_13TEV_LJ_DIF/uncertainties_dSig_dyttBar_interspectra.yaml ATLAS_TTBAR_8TEV_2L_DIF/uncertainties_dSig_dyttBar_norm.yaml ATLAS_TTBAR_8TEV_LJ_DIF/uncertainties_dSig_dmttBar.yaml ATLAS_TTBAR_8TEV_LJ_DIF/uncertainties_dSig_dmttBar_norm.yaml ATLAS_TTBAR_8TEV_LJ_DIF/uncertainties_dSig_dpTt.yaml ATLAS_TTBAR_8TEV_LJ_DIF/uncertainties_dSig_dpTt_norm.yaml ATLAS_TTBAR_8TEV_LJ_DIF/uncertainties_dSig_dyt.yaml ATLAS_TTBAR_8TEV_LJ_DIF/uncertainties_dSig_dyt_norm.yaml ATLAS_TTBAR_8TEV_LJ_DIF/uncertainties_dSig_dyttBar.yaml ATLAS_TTBAR_8TEV_LJ_DIF/uncertainties_dSig_dyttBar_norm.yaml CMS_1JET_8TEV/uncertainties.yaml CMS_2JET_7TEV/uncertainties.yaml CMS_2JET_8TEV/uncertainties.yaml CMS_TTBAR_13TEV_LJ_DIF/uncertainties_d2Sig_dyttBar_dmttBar.yaml CMS_TTBAR_13TEV_LJ_DIF/uncertainties_d2Sig_dyttBar_dmttBar_norm.yaml CMS_TTBAR_13TEV_LJ_DIF/uncertainties_dSig_dmttBar.yaml CMS_TTBAR_13TEV_LJ_DIF/uncertainties_dSig_dmttBar_norm.yaml CMS_TTBAR_13TEV_LJ_DIF/uncertainties_dSig_dpTt_norm.yaml CMS_TTBAR_8TEV_LJ_DIF/uncertainties_dSig_dyttBar_norm.yaml CMS_WPWM_13TEV_ETA/uncertainties_WM.yaml CMS_WPWM_13TEV_ETA/uncertainties_WP.yaml CMS_WPWM_8TEV_MUON/uncertainties.yaml CMS_Z0J_8TEV/uncertainties_PT-Y.yaml CMS_Z0J_8TEV/uncertainties_PT-Y_sys_10.yaml CMS_Z0_13TEV/uncertainties_AFB.yaml CMS_Z0_7TEV_DIMUON/uncertainties.yaml H1_1JET_319GEV_290PB-1_DIF/uncertainties_norm.yaml H1_1JET_319GEV_290PB-1_DIF/uncertainties_wo-lumi.yaml H1_1JET_319GEV_351PB-1_DIF/uncertainties_wo-lumi.yaml H1_2JET_319GEV_290PB-1_DIF/uncertainties_norm.yaml H1_2JET_319GEV_290PB-1_DIF/uncertainties_wo-lumi.yaml H1_2JET_319GEV_351PB-1_DIF/uncertainties_wo-lumi.yaml HERMES_NC_7GEV_ED/uncertainties.yaml LHCB_DY_8TEV_MUON/uncertainties.yaml LHCB_WPWM_8TEV_MUON/uncertainties.yaml LHCB_Z0J_13TEV_2022/uncertainties_dimuon_pT.yaml LHCB_Z0_13TEV/uncertainties_dielectron.yaml LHCB_Z0_13TEV_2022/uncertainties_dimuon_y.yaml LHCB_Z0_8TEV_DIELECTRON/uncertainties.yaml LHCB_Z0_8TEV_MUON/uncertainties.yaml STAR-2009_1JET_200GEV/uncertainties_CC.yaml STAR-2009_1JET_200GEV/uncertainties_CF.yaml STAR-2009_2JET_200GEV/uncertainties_A.yaml STAR-2009_2JET_200GEV/uncertainties_B.yaml STAR-2009_2JET_200GEV/uncertainties_C.yaml STAR-2009_2JET_200GEV_MIDRAP/uncertainties_OS.yaml STAR-2009_2JET_200GEV_MIDRAP/uncertainties_SS.yaml STAR-2012_1JET_510GEV/uncertainties.yaml STAR-2012_2JET_510GEV/uncertainties_A.yaml STAR-2012_2JET_510GEV/uncertainties_B.yaml STAR-2012_2JET_510GEV/uncertainties_C.yaml STAR-2012_2JET_510GEV/uncertainties_D.yaml STAR-2013_1JET_510GEV/uncertainties.yaml STAR-2013_2JET_510GEV/uncertainties_A.yaml STAR-2013_2JET_510GEV/uncertainties_B.yaml STAR-2013_2JET_510GEV/uncertainties_C.yaml STAR-2013_2JET_510GEV/uncertainties_D.yaml STAR-2015_2JET_200GEV_MIDRAP/uncertainties_SS.yamlAll of these uncertainty files are built using
covmat_to_artuncordecompose_covmat. These in turn rely on `numpy.linalg.eig' which leaves the evals and evecs unordered, most likely causing random fails.Changes
This PR introduces ordering of eigenvalues and eigenvectors in util functions
covmat_to_artuncanddecompose_covmat. It is as follows: eigenvalues are given in decreasing order, and the first non-zero entry of each eigenvector is positive.I also fixed filters that used
numpy.linalg.eigrather than our utils.Potential to do:
covmat_to_artuncanddecompose_covmatnecessary? They seem to do the same thing.covmat_to_artuncdefined twice (inutils.pyandcorrelations.py)?